Apple is changing the game in immersive video technology with its latest innovation for the Apple Vision Pro headset. Unlike traditional virtual reality (VR) and augmented reality (AR) devices that rely on standard fisheye or 360-degree video projections, Apple has developed a distinctive method that optimizes video quality and viewer immersion.
Typically, immersive videos in VR and AR settings are generated using fisheye lenses to capture wide-angle shots, which are then stitched together to create a spherical video environment. The Apple Vision Pro, however, supports these formats but also goes a step further by utilizing a unique streaming technology particularly tailored for its high-resolution capabilities.
Research spearheaded by Mike Swanson and released this Sunday reveals that Apple employs an unusual 45-degree rotation in its video encoding process. This strategy not only maximizes the pixel density along the most viewed areas, such as the horizon line, but also minimizes wasted space by eliminating unnecessary black sections often present in traditional fisheye projections.
Furthermore, Apple's method involves using an 180-degree equirectangular projection that stretches the image to cover the entire rectangular frame, enhancing the detail and quality of the visuals at the edges. This technique ensures that every pixel serves a purpose, enhancing the overall viewing experience by providing clearer and more detailed imagery.
Swanson's investigation also highlighted the sophisticated encoding used by Apple, featuring multi-view HEVC (MV-HEVC) which allows different video layers to be stored separately within the file, enhancing the efficiency and quality of the stream. Interestingly, Apple has chosen to apply this encoding technique not just to its proprietary content but also in streaming Apple TV+ videos, although third-party usage remains limited due to lack of documentation.
Despite these advances, some aspects of Apple's video projection techniques remain shrouded in mystery, such as the specific logic behind the minimal radial stretching and the strategic use of geometry to possibly deter copying of its format. As the tech community eagerly awaits further details, which may be unveiled at WWDC 2024, Apple continues to set new standards in the immersive video landscape.
